Deep Network Approximation for Smooth Functions

This paper establishes the (nearly) optimal approximation error characterization of deep rectified linear unit (ReLU) networks for smooth functions in terms both width and depth simultaneously. To that end, we first prove multivariate polynomials can be approximated by ReLU $\mathcal{O}(N)$ $\mathcal{O}(L)$ with an $\mathcal{O}(N^{-L})$. Through local Taylor expansions their network approximations, show $\mathcal{O}(N\ln N)$ $\mathcal{O}(L\ln L)$ approximate $f\in C^s([0,1]^d)$ a nearly $\mathcal{O}(\|f\|_{C^s([0,1]^d)}N^{-2s/d}L^{-2s/d})$. Our estimate is non-asymptotic sense it valid arbitrary specified $N\in\mathbb{N}^+$ $L\in\mathbb{N}^+$, respectively.